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ABSTRACT 

This handbook translates the findings and 
recommendations from earlier reports on purposes and practices in 
selected arts and sciences majors into a practical framework for 
reviewing the effectiveness of major programs in the arts and 
sciences as liberal learning. The handbook stresses the value of 
assessment as a dimension of program review and suggests ways cf 
"normalizing" assessment as an ongoing part of curricular 
expectations and students' learning programs. Following an 
introduction on the dynamics of program review, the second of the 
handbook's four chapters, "Key Elements of Strong Programs," 
discusses 13 characteristics of such programs: (i) clear and explicit 
goals, (2) focus on inquiry and analysis, (3) development of critical 
perspective, (4) connections with students' needs, (5) connections 
With scholarly inquiry, (6) connections within the major program, (7) 
connections with other disciplines and fields, (8) connections with 
liberal learning, (9) supportive community, (10) inclusiveness, (ll) 
advising, (12) evaluations and assessment, and (13) administrative 
supports, rewards, and recognition. Chapter 3, "The Program Review 
Process," discusses the role of assessment and review and steps in 
organizing the review. Chapter 4, "A Framework for Program Review," 
offers a series of questions organized around the 13 key 
Characteristics listed in Chapter 2. A selected bibliography cites 24 
resources and publications. (JB) 
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FOREWORD 



The Association ot American Colleges is pleased to share with colleges and uni- 
versities this new publication, Program Review and Educational Quality in the 
Major. Designed to complement standard approaches to program review, this 
handbook provides faculty members and academic administrators with a practi- 
cal framework for examining the educational quality of major programs in arts 
and sciences Helds. 

This handbook is the third volume in AAC's series on Liberal Learnmgand 
the Arts and Sciences Major. The first two volumes in this series— The Challenge of 
Connectinji Leaminfi and Reports from the Fields (1991 )-report on a national re- 
view of purposes and practices in selected arts and sciences majors. Conducted 
from 1988 through 1991 in collaboration with twelve national learned societies, 
AAC's national review examined campus practices in college majors and pro 
posed ways of reconceiving majors to strengthen their effectiveness as liberal 
learning. 

The Challenfie of Connecting Learning, written hy members of the National 
Advisory Committee tor AAC's project, offers standards tor liberal learning in 
any arts and sciences field. Reports from the Fields summarizes the twelve learned 
societies' recommendations tor improving learning in particular majors, 

This new handbook translates the findings and recommendations from 
these earlier reports into a pracrical framework for reviewing the effectiveness of 
major programs in the arts and sciences as liberal learning. It also draws on an 
earlier AAC project in which faculty members in fifty arts and sciences depart- 
ments in eighteen colleges and universities experimented with ways of assessing 
students' learning across courses in arts and sciences majors. Incorporating both 
participant and external evaluations of that study, this handbook urges the im- 
portance of assessment as a dimension of program review and suggests ways of 
"normalizing" assessment as an ongoing part of curricular expectation and stu- 
dents' learning programs. 

A NEW FRAMEWORK FOR PROGRAM REVIEW 

The initial idea tor this handbook came from faculty members and administra- 
tors who attended AAC's 1991 Annual Meeting and reviewed AAC^'s reci^mmen- 
dations tor strengthening liberal arts majors in the first two volumes of this 
series, which were released at that meeting. "If you really want faculty members 
to apply these recommendations," AAC members said, "link them ti^ institu- 
tional requirements tor program review. Tell us how program review should 
change." 

Following this advice, AAC convened an ad hoc advisory group and asked 
John Thi^rpe, a member of the Narional Advisory Committee for AAC's earlier 
project, to serve as the group scribe and handbook author. Members of that ad 
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hoc group examined program review protocols from several different colleges 
and universities and several learned societies. 

Few of these protocols raised searching questions about undergraduates' ex* 
perience of the major. None made the quality of learning and achievement in the 
major a primary focus. Only one addressed connections between students' learn* 
ing in the major and general education. None mentioned assessment of stu* 
dents' cumulative learning across courses as a dimension of program review. 
Only in the sciences were questions raised about students' learning in cognate 
subjects. 

This handbook complements rather than supplants prevailing approaches 
to program review in arts and sciences fields. Drawing especially on The Chal- 
lenge for Connecting Learning's framework for liberal learning in any arts and sci* 
ences field, this handbook invites examinination of the major's success in: 

□ shaping inclusive and supportive learning communities 

i J providing coherent and well^structured frameworks for learning 

□ fostering critical reflection on the field's values, presuppositions, approaches, 
and constraints 

□ supporting students' integration of their learning within and beyond the 
major, in and out of school. 

With these emphases, this handbook distills the insight gained over the past 
decades from a broad range of studies and experiments desigi.^d to strengthen 
the quality of both college programs and undergraduate learning. Tlie impor- 
tance of community as a support tor learning has been documented in many 
studies, most visibly but not uniquely in Philip Uri Treisman's research on ways 
of increasing minority students' success with calculus. Similarly, integrative cur- 
ricula are positively associated with college students' gains in both content 
knowledge and cognitive skills. In their meta-survey. Hot*/ College Affects 
Students: Findings and Insights from Twenty Years of Research (San Francisco: 
Jossey-Bass, 1991), Ernest T. Pascarella and Patrick T. Terenzini make this point 
succinctly: 

[Intellectual development) in a wide variety of areas is stimulated by 
academic experiences that purposefully provide for challenge and/or 
integration (A) curricular experience in which students are re- 
quired to integrate learning from separate courses around a central 
theme appears to elicit greater growth in critical thinking than does 
the same curricular experience without the integrative requirement, 
(page 619) 

There also is substantial evidence that formal learning is more effective and 
more likely to affect students' modes of thinking when it connects with their in- 
formal learning out of school. Otherwise students-even very able students— may 
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"play the game of school," using school learning to solve school problems while 
their mental models for outot-school issues remain largely unaftected by formal 
instruction. 

This handbook also reflects emerging wisdom from the assessment "move- 
ment/* As Alexander Astin and others have noted with increasing urgency, it 
serves little purpose to demonstrate whac students have learned or not learned 
unless the evaluator can connect that learning to important features ot the learn* 
ing environment. As program reviewers carefully observe how a program works— 
in fact as well as in design-while also studying findings from assessments ot 
students' learning, they can offer new insights on changes likely to make a signifi- 
cant difference in the quality of students' liberal learning. 
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INTRODUCTION 

CHANGING THE DYNAMICS 
OF PROGRAM REVIEW 



Periodic program review is a common practice in most colleges and universities. 
For some academic programs, particularly preprotessional programs, such review 
is part of an accreditation process conducted by an external agency. For other 
programs, including most undergraduate degree programs in the liberal arts and 
sciences, review is initiated by campus administrators as a means of monitoring 
program quality and identifying problems that may require administrative action. 

Often, program review in reality is a "program audit." The main focus of the 
review is on data: number of students, number of faculty members, student/fac- 
ulty ratios, qualifications of faculty members, class size, library holdings, scores ot 
seniors on standardized tests such as the Graduate Record Examination, percent- 
age of students who are admitted to graduate schtwl, percentage of students who 
are employed within a fixed period after graduation, percentage of classes taught 
by graduate students or by adjunct faculty members, percentages of women and 
minorities on the faculty and among the students. 

Important though these data may be. they are not sufficient to capture ade- 
quately the educational quality of a program. To assess educational quality, one 
must examine curricular and pedagogical issues— issues that are tied more to in- 
tentions, instructional practices, faculty/student interactions, and learning out- 
comes than to data. From this perspective, the^oal of a program revien' should he 
to increase the self<onsciousness of faculty members and administrators about their 
educational practices so ihey can improve the quality of teachinj^ and learning. 

This handbtx^k is intended for those who wish to initiate a program review 
focused on educational quality. Although a "program audit" r>-pe of review could 
be combined with the type of review advocated here, supplementary questions 
eliciting those data either can be constructed easily by anyone who wishes to in- 
clude them or adapted from one of many available prototypes. Therefore, "audit" 
questions will not be included here. 

The quality of education review, for which guidelines are provided here, is 
consistent with the national movement toward outcomes assessment. In the 
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end, one cannot truly determine the quality ot a program without understanding 
its effects on students. Thus, a program review that focuses on quality ot educa* 
tion depends on a clear specification ot desired educational outcomes and care* 
fijlly gathered evidence that these outcomes are being achieved, A variety of 
possible approaches to gathering such evidence are included in this handbook. 

The guidelines tor program review provided here address **majors"— pro- 
grams of concentrated study that in most institutions constitute the intellectual 
center for students' undergraduate learning. The guidelines retlect the findings 
of a national review of arts and sciences majors that the Association of American 
Colleges conducted from 1988 t(^ 1991 , The recommendations from that review 
are contained in the first two volumes of AAC's report on Liberal Learning and 
the Arts and Sciences Major, The program review guidelines presented here as 
Volume Three draw especially on Volume One of that report, The Challenge of 
Connecung Learning, which provides organizing principles tor liberal learning in 
any arts and sciences field. 

The guidelines presented here also retlect the conclusions of an earlier AAC 
project, Using External Examiners to Assess Student Learning in the Major, 
Participants in that project recomniended that program reviews begin to incorpcv 
rate findings from assessment of student learning. They also recommended that 
program review, which typically invites scrutiny from faculty members external 
ti^ the pn^gram (and often to the institution) ask such external examiners to re* 
view direct examples of students' learning across the major as part of rbei** over- 
all review of program quality. Suggestions for linking assessment to pfnoil'x 
program review are presenteil in this handbook- 

The second chapter of this handbook presents a philosophical tri.mework 
for program review. It is followed in Chapter Three by sugge^:tior i tor specific 
procedures that may be utilized in organizing a program review process. Chapter 
Four presents questiims to be asked in a program review process that is con- 
cerned primarily with the educational quality of learning in the major. 
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AAC*s 1991 report, The Chalkn$fc of Connecting Learning, advocates viewing un- 
dergraduate degree programs— programs ot concentrated study or academic ma- 
jors-from a strongly learnerK^riented perspective, paying special attention to the 
needs and expectations of students. This is not to say that students should deter- 
mine the content of programs— that must be a faculty responsibility— but rather 
that faculty members, as they design, implement, and review programs, should 
be sensitive to the students ser\'ed by the program. This sensitivity cannot be left 
to one or two faculty members who have a particular interest in undergraduate 
students; it must be shared by all who teach in the program. The following thir- 
teen characteristics of strong programs, articulated in The Challenge of Connecting 
Learning, provide the underlying philosophy that guides AAC's approach to prcv 
gram review. 

1. CLEAR AND EXPLICIT GOALS 

It is not enough for deliberations about the major to be exercises at the 
blackboard diagramming curricula that ''look right'* but have little effect 
either on course practices or on student experience of the major. Faculty 
members' deliberations about majors as educational programs need to be- 
come part of a continuing collegial dialogue about the relationship between 
/acuity intentions and student progress. 

The Challenge of Connecting Learning, page 21 

Strong programs articulate clear goals for students* learning, which are made ex- 
plicit and understandable to the students. These goals should include clear ex- 
pectations about the purposes and character of introductory, middle-level, and 
culminating work; the nature of and rationale for program requirements; and the 
rationale for curricular structures as they relate to these goals. 

Faculty members in each program must accept corporate responsibility tor 
the program's goals and curriculum. Tlie faculty as a body periodically should re- 
view the program's goals, the extent to which they are realized, and their relation 
to the goals and structures of general education. Ideally, the faculty will consider 
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thoughtfiilly, and explain carefully to students, how the articulated Koals tor 
learninsland curricular structure) in the program relate to the overarching goals 
for students* learning in the institution as a whole, including general education. 

A FOCUS ON INQUIRY AND ANALYSIS 

The problem U'lth the maior is not that it has failed to deliver eertain kinds 
of knowledge. The prohlcm is that it often delivers too much knowledge 
with too little attention to hoiv that knowledge is being created, U'hat meth- 
ods and modes of inquiry are employed in its creation, u'hat presupposi- 
tion: inform it, and what entailments flow from its particular ways of 
knowing 

Students have the right to expect their major to provide a set of learn- 
ing ex/>eriences that will teach them how to use their field *s approaches in 
pursuing significant questions. . . . They hare the right to expect learning 
experiences that ivill encourage them to shape, reflect on. add to, and use 
the Icnoivletise they are gaining. 

The Challengeof Connecting Learning, 6-7 

Strong programs help students develop the capacity to use the methods and per- 
spectives of the discipline(s) in framing questions and in developing increasingly 
sophisticated analyses of those questions. Recognizing that these capacities de- 
velop over time, these programs create curricular structures that provide stu- 
dents with opportunities to revisit issues that they have met in prior courses and 
to bring to bear on those issues increasingly powerful analytic techniques. 

3^ DEVELOPING CRITICAL PERSPECTIVE 

fStwtlentsj have the right to expect opportunities for translating and negoti- 
ating among different approaches and for explorini^ the strengths and limi- 
tations of the lenses through which they have learned to view issues and 
problems. 

The CuALLENciEeiF Connecting Learning, 7 

Students }oin the community of the major briefly; xdtimately, they must dis- 
engage and leave. An essential step in this process of disengagement is the 
achievemevt of some measure of critical distance. Part of the articulated 
purpose of the major, therefore, is to prepare a student to be sufficiently 
confident in the discourse of a community to subject the major to sophisti- 
cated cjuc'stions and to compare and connect its proposals with the propos- 
als of other communities. 

The Challenge OF CoNNbCTiNG Learnincj. 12 
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Strong pmgrams help students understand the limitations of a particular field's 
methods or pevspectives by providing opportunities for students to consider the 
values, biases, and internal conflicts of the field and to examine the ways in 
which the field may be inadequate (at least on its own) tor dealing with some 
kinds of questions. In some fields, this may require that faculty members engage 
in more selt<onscious analysis of their discipline than they are accustomed to 
doing. 

Strong programs help students learn about the variety ot views and perspec- 
tives represented within each field. Engaging students in open discussion ot ques- 
tions and conflicts about the presuppositions, methods, and findings ot the field 
represents a powerfi.il teaching technique that fosters intellectual development. 

4. CONNECTING WITO STUDENTS* NEEDS 

Faculty members often think of the major as a study of a subject valuable 
in itself, or as a preparation for advanced, postbaccalaureate studies (with 
the desire that the best students themselves should enter the professoriat). 
Students often speak of attaining usable capacities, of the "real world'' 
value of collegiate education. The fact is that most students do not go to 
graduate school and a career in the learned professions, ncrdo they use the 
content of their major directly in their careers. 

The Challenge OF Connecting Learning. 3 

Programs must acknowledge and respect the perceptions and expectations ot stu- 
dents who choose to pursue a particular course of study. This implies that the 
faculty must he aware of the characteristics of the student body. \X/hat are the stu- 
dents' backsrounds. both academic and experiential? What are their aspirations? 
What do they believe about the value of a college education and about the role ot 
the program in that education? What questions and concerns about the tield are 
of greatest personal interest to them? 

Students change; society changes; career opportunities chanjje. Strong pro 
Krams are dynii mc, not static. Faculty members in strong programs actively and 
regularly seek to learn about their students and respond appropriately to their 
chan^inK needs. Sometimes the program, or parts of it, must be retormed. 
Sometimes, extra efforts must be directed to helpinj^ students understand more 
tiilly the Koals of the profjram as conceived by the faculty. Sometimes, taculty 
members must articulate the profjram more careflilly to counteract— beforehand- 
misplaced student assumptions. Always, keen faculty awareness of student per- 
spectives and willingness to rethink their own convictions in the liyht ot new 
understandings about students are necessary prerequisites to a healthy program. 
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5^ CONNECTING WITH SCHOLARLY INQUIRY 

[There is] a marked disparity hetiveen the ivays academics do their re- 
search and the institutional structures that organize curricula and teach- 
ing. . . . What are needed are incentives and structures to ensure that the 
intellectual excitement of discovery, interaction, and critical discourse that 
man;y faculty members experience also is available to students. 

The Chal!,enge OF Connecting Learning, 15 

Strong programs reflect the current state ot scholarly understanding. As ad- 
vances are made in the tield, they should impact on the curriculum. As new 
ideas emerge tor organizing and communicating knowledge in the discipline, 
they should influence the structure and pedagogy of the program. As new issues 
and controversies arise that relate to the content of the program, they should be 
brought into the classroom. 

Faculty members must be active in the field if they are to design and teach a 
relevant and vital curriculum. They must be well qualified ininally, and they 
must maintain an acrive regimen of scholarship. Some will be researchers or cre- 
ative artists. Others may focus on interprenve, expository, or teaching and learn- 
ing issues. All, however, must belong to the community of scholars: learning, 
exploring issues within a coUegial community, and sharing new understandings 
with students. All should have explicitly stated personal goals for professional 
development and scholarship and for improvement as teachers. Moreover, just 
as program quality should be reviewed periodically, so should each foculty mem- 
ber's progress toward his or her stated goals be reviewed regularly by peers. 

6. CONNECTIONS WITHIN THE MAJOR PROGRAM 

Majors. . . require faculty members' willingness to develop a shared under- 
standing o/u'ha( study in the major is supposed to accomplish and faculty 
members * collaboration in designing a coherent program of study sufficient 
to accomplish it. 

The Challenge of Connectincj Learnincj, 7 

The program should be organized around a carefijl plan that views it as a coher- 
ent whole rather than as simply a collection of courses. In some programs, coher- 
ence is achieved partially through prerequisite structures. In others, it is achieved 
partially through increasing specialization or methodological sophistication. The 
need for establishing connections goes beyond either of these, however. A strong 
program will include opportunities for synthesis and integration across courses- 
perhaps through junior or senior seminars, a capstone course, or a senior thesis. 
Faculty members must be willing to accept responsibility for the enure pro 

o 15 

ERIC 



KEY ELEMENTS 
OF STRONij PROCjRAMS 



Uram— not just tor their own courses* They must work collaboratively to ensure 
that connections among courses are visible to students* This can happen an\y it 
faculty members know what their colleagues are teaching and how their own 
courses tit within the context ot the program* 

?• CONNECTIONS 
WITH OTHER DISCIPLINES AND FIELDS 

To fuifill jt5 role in liberal learning, the major also must strueture conver- 
satiorxs with the other cultures rL*()re.SL»ntL»ci in the academy, conversations 
that more nearly reflect the diversities mthin our world and require patient 
labors of translation. Ultimately, the goal of the major should be the devel- 
opment of students'capacities for making connections and for neneratinR 
their own translations and syntheses. 

The Challenge OF CoNNtcTiNu Llarnincj, 5 

At its best, the traditional major has offered a curriculum desif^ned to con- 
vey what is central to a given discipline or area of study. But the svnthesi^- 
ing ertterprise-the bringing of what one has learned in one context to 
another, from one communit\ to another-has been left almost entireN to 
students' private initiative. It ought to take place in public, accredited, cur- 
ricular space. 

The CHALLENGt OF Connecting Learninc^ 15 

Strong programs help students see and appreciate the connections with other 
disciphnes that faculty members may take for granted. Such seeing and appreciat- 
ing can he accomplished through interdisciplinary courses, seminars, or projects; 
through course requirements outside the major department; or, perhaps hest, 
through a continual effort hy faculty members to examine in their own courses 
insights into, examples from, and applications to other fields. 



8* CONNECTIONS WITH LIBERAL LEARNING 

ITjhe traditional distinctions between general education and the major no 
longer can be sustained. . . . ITjhe work of the major needs to open into a 
larger context of learning in order to develop the fullness of perspeetii'e that 
the discrete disciplines and fields of study cannot help but obscure. 
TheChallenoeofConnectincj LEARNINCj, 5 

Strong programs acknowledge and tea::h connections between issues ot their 
held and issues of wider academic and "real-world" ci^ncern. Every practitioner 
in every field regularly encounters political, societal, and ethical issues related to 
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the field. Attention to these issue , should not be confined to general- education 
courses. Students mus grapple with some ot these issues within the context ot 
the major. Students aiso need opportunities to explore the connections between 
the various topics and issues in the field and their own interestj> and concerns. 

Each program must accept responsibility tor a role in the general education 
of its students. Strong programs may require, tor example, a course on echics in 
the discipline, a seminar that examines case studies ot major advances in the field 
and includes attention to the broad issues associated with those advances, or an 
independent study course that permits the student to explore in depth some top- 
ic in the field that is personally meaningful. They may create capstone courses 
that encourage topical connections across fields. 

9. SUPPOPTIVE COMMUNITY 

Faculty members must take seriously what students believe about a given 
subject and engage their prior knoijuledge so that neu' learning restructures 
the old, complicating and correcting it rather than merely living side by 
side ivith it. 

The Challenge OF Connecting Learning, 13 

Strong programs provide practical support and encouragement for students' in- 
tellectual growth and development. This includes providing frequent opportuni^ 
ties tor interaction and dialogue among students, and between students and 
faculty members, in addition to that which occurs in the classrtK^m. 

The faculty members in a program must take a special interest in the intellec- 
tual development of the students in the program. This responsibility can be met 
only when faculty members recognize that intellectual development is a scKial as 
well as an individual process. Program faculty members must provide frequent 
occasions that enable students to engage with others in a collaborative explora- 
tion of knowledge and inquiry in the field of study. Faculty members must be 
concerned with the many factors that afi'ect students' lives as learners, and they 
must try to provide an environment in which learning is as natural, stimulating, 
and fijlfilling as possible. 

10. INCLUSIVENESS 

Redressing imbalances (among students due to age, ethnicity, economic 
background, gender, and race] cannot be left to the admissions office or to 
an institutions promising collaboration vuith the local public schools. 
Faculty members in tach program must explore a hat obstacles their fields 
present to the participation of discrete groups of underrepresented students 
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and make a strong commitment to eliminating those obstacles. 

The problem of full ()artici()ation in arts and sciences majors no 
longer can he framed in terms of access alone; U'hat is needed is a re/orma- 
tion 0/ present practices. 

The Challenge OF Connecting Learning, 17 

Tliose responsible tor strony programs are aware ot the demoKraphic shifts in 
representation ot minorities and women in the workplace and the benefits to 
their department ot seeking to serve students with diverse characteristics and 
backgrounds. They work actively to tind ways to attract and retain students otall 
types and backgrounds, including those who have been discouraged from partici- 
pating in the tield due to stereotyping and bias. 

Faculty members in strong programs work to improve access to and persis- 
tence in the program amt^ng groups that are underrepresented in the tield. They 
approach this work with the recognition that focusing on the diverse back- 
grounds, learning styles, and concerns ot new groups of students will result in 
improved learning experiences and educational quality for all students in the 
program. 

Those responsible for strong programs are alert to the possibility that implic- 
it and unintended messages may be communicated that suggest to students that 
they are not welcome. They structure introductory courses to acknowledge and 
respect the goals, perspectives, learning styles, and experiences of all the students 
in those courses. They take particular care not to use introductory courses as fil- 
ters that discourage students who in fact have potential ft^r success in the field. 

Strong programs provide multiple points of entry. These points of entry 
may include programs for precollege students that encourage them to enter the 
field, prt)grams for tirst-year students whose academic records indicate they are 
capable of success in the field in spite of stereotypes that may have discouraged 
them, and courses— or versions ot courses— that acknowledge the differing prepa- 
rations and interests of students. 

Strong programs acquaint students and potential students, through orienta- 
tions or units in Hrst-year courses, with the culture ot the tield and work actively to 
make them feel at home. Strong programs actively seek to recruit a diverse taculty 
that represents the scholarly attainment of traditionally underrepresented groups. 

Faculty members and administrattirs in strong programs seek alliances with 
ethnic and gender studies programs, such as American ethnic studies and wom- 
en's studies, to offer joint courses or programs on the history and contributions 
of these groups to particular fields. They participate in seminars tt) acquaint 
themselves with new scholarship. Where appropriate, they incorporate knowl- 
edge gathered from these kinds of alliances and inquiries into their own courses 
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and proKrams. They also acquaint themselves with the nature of racial, ethnic, 
and sexual harassment and with inadvertent biases that may manifest them- 
selves in teaching. 

11. ADVISING 

Careful advising, is. . . necessary, not only to assure that the requisite num- 
ber 0/ courses are taken but also to support a continuing discussion about 
the program's purposes and students' experiences. 

The Ch.alleni-.e OF Connecting Learning, 11 

Advising students is a faculty responsibility. Advising should befjin as soon as a 
student shows a potential interest in the program, and it should continue 
through graduation. Hifzh-quality advising includes sharing information on the 
goals and expectations of the program; helping students devise purposeful and 
coherent plans for study; discussing the culture of, and opportunities in, the 
Held; exploring the aspirations and interests of the student; discussing employ- 
ment opportunities in the Held; and assisting students in the selection of gradu- 
ate schotils. 

Throughout the process, advising should help the student successfully "ne- 
gotiate the system," ensuring that the college experience is as smooth and pro 
ductive as possible. High-c|uality advising includes encouraging and supporting 
.student activities outside the formal curriculum, such as academic clubs, student 
seminars, and interest group.s. as well as the traditional one-nn-one faculty/ stu- 
dent consultations. 

12. EVALUATION AND ASSESSMENT 

fSJtucIents have the right to expect that all of the capacities and Icnou'ledg^' 
they have gained uill be assessed, by /acuity members, through carefully 
designed occasions that challenge them to integrate and demonstrate their 
learning across their specific programs 0/ study 

The Challenge of Connecting Learning, 7 

Strong programs incorporate occasions tor assessing student learning that tran- 
scend individual courses, tor marking educational milestones in the program, and 
tor evaluating cumulative learning. This kind of assessment is essential if faculty 
members truly are to know how eftective their instruction is and how well the 
goals of the program arc being met. Thoughttxjlly and caretully done, such evalua- 
tion also can enhance the process of helping students to make connections: con- 
nections between courses in the program, connections with other tields of study, 
and connections with their own persv)nal experiences and interests. 
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Although faculty members initially often are suspicious ot— and resistant to— 
assessment, the benefits of assessment usually are recognized quickly by faculty 
members who engage in assessment activities. Assessment allows faculty mem- 
bers to identity what they are doing right-what works well and what succeeds in 
involving students more actively in the Held. The results of assessment allow fac- 
ulty members to streamline their work, abandoning classroom practices and ac- 
tivities that do not significantly enhance learning. 

Outcomes assessment is a necessary part of program review. Analysis of in- 
put alone (design of the curriculum, number and qualifications of faculty mem- 
bers, class size, quality of classrooms and laboratories, library holdings, and so 
on) cannot yield information on the effectiveness ot programs. 

13* ADMINISTRATIVE SUPPORT, REWARDS, 
AND RECOGNITION 

This report. . . asks for collective and collaborative faculty discussions 
about ways 0/ translating these common commitments into institutional 
practices and structures. This translation must begin u'ith time and 
space. . . . ll]t must be supported by visible and concrete reuwds. . . . 

The Challenge OF Connecting Learning, 21 

Any program's hculty members will be committed to a diverse set ot activities in- 
cluding teaching, scholarship, course and curriculum development, advising, 
evaluation, and assessment. The quality of the program depends on strenf,th in 
all of these areas. This does not mean that every tacult>^ member in the program 
personally must be committed to excellence in each of these areas. It does mean, 
however, that all faculty members must acknowledge the importance of each of 
these areas; that tor each area there are faculty members in the program who de- 
vote significant time and energy to the area; and that excellence is recognized, ap- 
preciated, and rewarded. 

Strong programs have clearly articulated policies for supporting and reward- 
ing all of these types of activities. Rewards include salary increases, substantial 
weight in promotion and tenure decisions, and awards in recognition of excel- 
lence. Administrative support includes timds for curriculum development and 
faculty development (for example, workshc^ps and consultations); for research 
and scholarly pursuits; tor course and teacher evaluation; tor assessment; and tor 
travel to meetings and conferences that deal with teaching, learning, and assess- 
ment, as well as scholarship in the Held. 
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In a very real sense, program review is (or should be) a continuous process. 
ThouKhttul facult>' members, as they teach and advise students, constantly reflect 
on the etiectiveness i^f what they are doiny and t^n the implications tor their courses 
ot what they are learning from their students. Ideally, they also reflect on the im- 
plications tor the program as a whole and share those retlections with other tacul- 
ty members in faculty meetings or perhaps a faculty retreat. The result is ccintinual 
fine-tuninK of courses and modification of the program whenever the evidence 
suggests such modification is needed. 

Pericxlically, however, a more structured review is necessary. These structured 
reviews should occur at regular intervals, not simply occasionally in response to a 
"crisis" perceived by a chair or dean. Although the stated purpose of the review 
may vary-to inform external constituencies about the quality ot the program, to 
focus attention on aspects of the program that need attention, or perhaps to jus- 
tify a request for additional resources— the ultimate goal of a program review 
should be to examine the extent to which the educational goals ot the program 
are still appropriate and are being achieved satisfactorily. Almost inevitably, a 
structured program revievv will result in some (possibly minor, sometimes major) 
changes in the program. The ultimate goal of any prc^gram review shtviilJ be im- 
proving the program. 

The frequency of structured program review also will vary greatly across in- 
stitutions and across programs. Typicallv, structured program reviews occur a[> 
proximately once every five to seven years, although special circumstances (tor 
example, a program in transition, a program on "probation," or an intervening 
accreditatit^n review) may indicate the need for a review after fewer than tive or 
more than seven years. 

The most usetiil program reviews are based on evidence that is collected on 
an ongoing basis, not just assembled immediately prior tt^ the review. Continual 
evidence gathering is impt>rtant to ensure the availability of the necessary kinds 
of evidence, the rt^bustness of data, and the ability to analyze trends. It also acts 
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as a reminder ot the importance of staying Dleit to factors that affect program 
quality. 

Program review always invites feedback from reviewers outside the immedi- 
ate program. Usually, these external reviewers will include colleamies from other 
institutions. External perspective is an important dimension of program review. 
It challentjes faculty members to reconsider their programs in relation to prac- 
tices at other institutions and it connects their work with a larger community ot 
dialogue and debate. 

ASSESSMENT AND PROGRAM REVIEW 

A key ingredient in successful program review is the quality ot evidence docu- 
menting the educational outcomes of the program. This evidence should assess 
the quality of student learning. It should be aggregate, focusing on the program 
and not on individual students or courses. It should be derived from multiple 
assessment strategies; no one approach can provide adequate information. 

Assessment should focus on students* growth from the time they enter the 
program tl\rough their graduation from it. It should provide evidence of the 
kinds of work students are attempting, their increasing capacity to synthesize and 
integrate knowledge from different parts of their program, and their ability to use 
the approaches of their field in framing and analyzing problems and issues. It 
should provide evidence of the ways they are using their studies and not just evi- 
dence of their knowledge of specific information. To serve the purpose of pro 
gram review, assessment need not include information on every student in the 
program; information from representative samples is sufficient. If the program 
has many transfer students, the assessment should provide evidence of how 
transfer students as a group are adapting to the program*s goals and expectations. 

Although assessment for program review requires only periodic sampling ot 
students* learning, such assessment is most informative when it is grounded in 
course and program requirements. Many campuses find that students do not do 
their best work when assessment is extracurricular and has rio consequences tor 
them. Similarly, faculty members may not value evidence developed only tor the 
periodic program review. They rnay view the evidence as irrelevant because they 
do not know and trust the measures. When assessment is embedded in the regu- 
lar curriculum, faculty members are better able to interpret the results and use 
them for program improvement. 

The best assessment strategies provide learning experiences for students as 
well as evidence for program review. Assessment can be embedded in courses by 
including common questions on examinations in several courses in the prtv 
gram. Assessment can provide advising experiences tor students through "rising 
junior" exams that assess learning outcomes through the junior year and provide 
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information to guide study during the senior year. Assessment can help students 
recognize and appreciate their own growth in the program through portfolios or 
self assessment essays. Assessment can be structured as a series of assignments 
in acourse— i^rset of courses-that allow students to try again if their performance 
is below expectations. Assessment can provide integrative experiences through 
senior projects. 

Program reviews also can incorporate assessment by external examiners. 
Qualified persons in the external program review group can explore, in oral inter- 
views, the extent to which students have appropriated concepts, approaches, and 
knowledge in their field. Tliey can evaluate how well students can synthesize or 
draw upon this learning in addressing new issues or problems. Such external re- 
views are most effective when the examiners base their questions on students* 
written work-for example, in a comprehensive examination, a senior project, or 
in a series of course papers. The results should be summarized in a written re- 
port to the program faculty members. When students present their work to ex- 
ternal examiners, the review process also should include feedback to the students 
themselves. 

(Jcher assessment strategies may include pre- and post-tesring (testing at pro 
gram entry and retesting at program completion), surveys of graduating seniors 
and of alumni/ae, exit interviews, analysis of alumni placement, tracking ot 
alumni, standardized tests, and transcript analyses. 

ORGANIZING THE PROGRAM REVIEW 

The first step in a structured program review usually is a selt^study. This selt- 
study should produce a report that reviews educational goals and rationales tor 
the program, summarizes conclusions of the previous program leview, describes 
changes in the program since the previous review, provides evidence indicaring 
the extent to which program goals are met, idenrifies critical problen\s facing the 
program, and includes short- and long-range plans and recommendations. 

The preparation of the self-study report usually is guided by a set of ques- 
tions that focus attention on issues considered important by the faculty and ad- 
ministration. A set of possible questions is provided in the next section ot this 
handbook. The self-srudy report should reflect the corporate judgment ot pro 
gram faculty me»Tibers and, ideally, will be the result of a collegial effort that in- 
volves the entire orogram faculty. 

A second step in most program reviews is an analysis of the self-study by a 
group external to the program. This group may include faculty members from 
other programs in the same insritution, faculty members from similar programs 
at other institutions, alumni, students, members ot advisory boards, administra- 
tors, or any combination ot the above. This group studies the report; reviews 
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course syllabi, final examinations, transcripts of recent graduates, and assess^ 
ment findings; interviews faculty members, students, and administrators asso 
ciated with the program; and requests any additional information deemed 
necessary. Members of the group also may conduct oral interviews with stu- 
dents about their work and experience in the program. Using this information, 
the group prepares its own report, which comments on the quality ot the selt* 
study as well as on the plans and recommendations contained therein. This 
group also will add its own recommendations. 

The final step in the program review process is the institutional and prcv 
gram response, which provides an occasion for constructive dialogue betu'een 
program faculty members and the responsible dean, as well as an opportunity tor 
program faculty members to revisit the analysis contained in the self-study. The 
main Kvcus of these discussii)ns should be on educational quality and should re^ 
turn once more to the basic questions: What are the educational goals? How 
well are they being achieved? What changes should be made in light of the re- 
view findings? A follow-up session with the dean should take place after an ap- 
propriate interval to discuss any changes that have occurred as a re3ult ot the 
program review. 

Program review, dt)ne well, is more than simply an activity carried out from 
time to time in response to a periodic need for information about the health ot a 
program. Program review should be a continual process: of goal identitication 
and review, i)f faculty' conversation about teaching and learning strategies, ot as- 
sessment and revision. The tbrmal evaluation, which includes the selt-srudy and 
the report of the external review committee, should be simply a signpost in that 
process, showing the results of prior assessment activities and pointing the way 
to a continuation of these activities in the years ahead. 
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An efficient and effective program review must be guided by questions that call at- 
tention to impcmant issues and suggest standards, and whose answers can pro- 
vide a basis for evaluation. The set of questions that guides a program review must 
be tailored to campus goals and priorities. No single set will be appropriate for all 
campuses or even tor all programs on a single campus. Nevertheless, although 
one can expect considerable variation, most program reviews will need to deal 
with similar basic issues that reflect areas of common concern. The questions pro- 
vided here t^ffer a model for campuses to use in constructing their own guidelines 
for program review. Admittedly and unapologetically, they reflect the views about 
strong programs of concentrated learning articulated in The Challenge ofConnea- 
ing Learning and summarized in Chapter Two of this handbot^k. 

The questions are grouped in categories. We recommend that each campus 
add one or two additional categories to the list-<ategories that deal with local 
concerns— and that the specific questions suggested here be reviewed carefully 
and modified as necessary to reflect campus priorities. 

GOALS 

IFacuhy members must aitempij to provide a local structure for a course of 
study in a major that can specify its goals, ensure that these goals are com- 
municated to students and faculty members alike, and assess the degree to 
ii'hich these aims are achieved. 

The ChallenciE OF Connecting Learning, 7 

Prior to any assessment activity there must be an idenrificarion of glials and ex- 
pected outcomes. In a strong pn^gram, these will be articulated prior ti^ program 
implementation. They will be under review conrinually and, from time to time, 
under revision. 

■ >X^at are the educational goals of the program? 
. How were these goals determined? 
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Are all faculty members in the program aware of the goals, and do they 
understand how their own courses are intended to contribute to achieving 
these goals? 

How are the goals of the program communicated to students? 
Is there clear evidence that the goals of the program are being met? 

■ Are the goals of the program appropriate tor the blend of faculty members and 
students that are in the program? 

How do the goals of the program compare with the g N of similar pro 
grams at other institutions? 

Are program faculty members aware of the characteristics of exemplary prtv 
grams at other institutions? 

■ What are the intended outcomes of the program? 

Are these outcomes slated in terms that permit judgments about the extent 
to which they are realized? 

^X^at procedures are in place for collecting and analyzing evidence that 
enables such judgments to be made? 

Does this evidence show that the intended outcomes are achieved? 

■ How does the program monitor its progress toward achieving its goals? 

Are faculty members involved in this process? 

Are occasions provided for sharing the results with all faculty members? 

■ >X^at are the major changes occurring in similar programs in other 
institutions? 

How does the faculty learn about these changes? 

How does the faculty assess which of these changes to implement locally? 

■ \Xliat process is used for regularly reviewing goals, courses, and curricular 
structures in light of the findings of assessment activities? 

Is this a corporate process that involves all faculty members in the program 
or at least a broadly representative cross'section of the faculty? 

What have been the results of these processes since the previous program 
review? 

■ What mcKiifirations have been made recently in the goals or in the program? 

Have these modifications resulted in documented improvements? 
Are there problems that have not been addressed? 

THE STRUCTURE OF THE CURRICULUM 

IFjaculiy members Imustj concern themselves not just ivith course require- 
ments but u'lth the ways that a major*s parts and practices contribute? to its 
larger purposes. . . . The chosen mode of organization. , . ought to be the re- 
sult 0/ deliberate and corporate faculty judgment. 

THECHALLENCitOFCONNECTINCi LBARNINei. IS 

or 
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In a strong program, required courses and constrained electives fit together in a 
coherent whole. The organization of the program will vary from field to field 
and, perhaps, fi^om institution to institution. For all programs, however, the or- 
ganizing principles and coherence of the curricular plan should be evident to stu- 
dents and faculty members alike. Faculty members should possess a shared 
understanding of the differences between introductory, intermediate, and culmi- 
nating work in the program. Their courses and assignments should reflect this 
iihared understanding, 

■ What is the plan tor the curriculum and how was it determined? 

Is it based on a well-defined intellectual agenda? 
. Is the plan understood by all faculty members teaching courses in the 
program? 
; Is it understood by students? 

Is it reflected in course rationales, syllabi, and assignments? 

■ Does the program begin with survey courses or with more specialized introduc- 
tions to the field? 

What is the rationale for this choice? 

Do the beginning courses serve both majors and nonmajors? 

If so, is there evidence that they seive both constituencies well? 

■ Is there structure in the middle range of courses? 

What are the organizing principles? 

Do these courses build significantly on the introductory courses? 
Do they acknowledge and utilize the learning that is occurring in other 
middle-range courses? 

■ Do the middle-range courses include attention to connections with other fields 
and with the learning that is occurring in the other parts of the curriculum (for 
example, in general-education courses)? 

Do they allow rime and space for reflecring, synthesizing, and generalizing? 

■ Are students introduced early to the modes of inquiry and methodology of the 
discipline? 

Are these modes and methods then utilized in assignments for subsequent 
courses? 

Do students exit the program with a demonstrated ability to apply the apv 
proaches of the discipline to formulate and analyze new questions, conjec- 
tures, and proposals? 

■ Do beginning or middle-range courses introduce students to the contested is- 
sues of the field and provide students the opportunity to engage actively with 
these issues? 

Do students understand both the strengths and the limitations of the 
methodology and perspectives of the field? 
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Do they exit the program with some measure ot critical distance trom the 
Held? 

■ Is there a common core ot courses taken by all students in the program- 

If not, is there an alternate academic structure that stimulates the develop- 
ment ot an intellectual community amon^ the students? 

■ Is there a capstone experience (tor example, a senior seminar, a senior project, a 
thesis, comprehensive examinations) that provides students with an opportunity 
to integrate the learning that has vKCUrred throughout their college experience? 

Does this capstone experience integrate courses external to the program— 
including courses in general education-as well as courses within it? 

Does it challenge students to grapple with some ot the ethical, political, and 
stKietal issues associated with the field? 

■ Wliat characteristics of the program are evident from an analysis ot student 
transcripts? 

Is the intended structure of the program realized? 

Is it erteciive in supporting the educational goals of the program? 

Has the structure been reviewed or modified recently? 

Does it need modification? 

CONNECTIONS 

IFjosterin^ capacities for reflection on what happens beyond the academy 
must be the larger goal. The discourse of the academy is hut a means to an 
end, a developmental step along a path that appropriately points students 
toward a multitude of contexts and circumstances. 

The Challenge OF Connecting Learning. 14 

It is. . . important for [students] to care about subject matter and see its im- 
plications for the ways they live their lives. At issue is whether students 
can connect a field's subject matter and approaches with a variety of pur- 
suits important to them, and tthether their curiosity and concerns beyond 
the classroom can be deepened or shaped by the insights the field brings 
forth. This recjuires teaching and opportunities for re/lection that encour- 
age students to test the assumptions and proposals of the /ieltl aji^ciinst ques- 
tions and evidence drawn from their own experience. 

The Challenge OF CoNNECTiNci Learning, 16 

Strong programs are designed to facilitate connections: with the most recent ad- 
vances in the field, with the practice of the field in lite beyond the academy, with 
applications to other fields, with liberal learning, and— most important from stu- 
dents' perspectives— with the needs and lives ot students. Inattention to these 
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connections will lead to an insularity that is certain to result in a loss of vitality 
for the p'-ogram. 

■ How important are these connections among the stated goals ot the program? 

To what extent are connections explicitly addressed in the curricular 
structure? 

Is there evidence that students are making connections as they progress 
through the program? 
As they exit the program? 

■ Whar are the modes of scholarship in which the program's faculty members ac- 
tively are engaged? 

Is there a good representation ot differing modes ot scholarship among the 
program taculty members? 
Are all faculty members active? 

Do they bring into the classroom the excitement as well as the results of re- 
cent work in the field? 

Do they regularly revise their courses to include the latest results, debates, 
and open questions? 

■ Do experiences provided in the program connect with tne principal career op- 
tions available to graduates of the program? 

Are there opportunities for internships or summer employment? 

■ Do faculty members pay attention in their courses to the links between their 
courses and the overall goals of the program? 

To the links between their discipline (or approach to the discipline) and 
others? 

Do they encounge students to take multidisciplinary courses and courses in 
other fields that extend or use the techniques, ideas, or content of their field? 

Does the program offer opportunities for interdisciplinary courses, semi- 
nars, or projects? 

■ How does the program curriculum intertace with the general-education 
curriculum? 

Do intermediate and advanced courses in the program utilize the knowledge 
that students bring to these courses from general-education courses? 

Do courses in the program attend to the social, political, and ethical issues 
associated with the field? 

Are there opportunities for students to connect what they are learning in 
the program with the wider issues of liberal learning and with issues that are 
important to them personally? 

Does the program provide opportunities for students to engage in a process 
of generalizing that reaches beyond the confines of the discipline and into a 
broader context? 
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■ How does the program connect with the lives ot students? 

Does the program actively seek to learn about the characteristics of its 
students? 

Is there evidence that the program is responsive to changes in those 
characteristics? 

In particular, is there a regular effort to assess the academic preparation that 
students bring into the program? 

Is there a regular effort to deteimine the students* aspirations, beliefs, and 
expectations with respect to the program? 

Is there evidence that the faculty members are aware ot these findings and 
acknowledge them in their courses? 

Has the program as a whole responded to these findings? 

TEACHING QUALITY 

For students, learning in the major means learning to take part in a con- 
tinuing ex()loration. The role of faculty members is to provide structures 
and languages that support this participation.... 

The Challenge OF Connecting Learning. 4 

Strong programs are taught by faculty members who place a high priority on the 
quality ot their teaching. Teaching quality involves much more than perfor- 
mance in the classroom. It includes attending to the needs ot students with a di- 
versify of interests, backgrounds, and learning styles. It includes interacting 
with students both inside and outside the classroom in relation to the overarch- 
ing learning goals of the undergraduate academic experience as well as to the 
goals of a particular course or program. It includes staying current on recent re- 
search in teaching and learning, and it includes a willingness to alter teaching 
styles on the basis of the results of that research. It includes responsibility tor ad- 
dressing the shared goals of the overall program in planning courses and in giv- 
ing feedback to students on their work and progress. 

■ How does the program encourage high<iuality teaching? 

Are incentives and rewards (such as teaching awards, salary increases, hjnds 
to attend conferences on teaching) provided to promote and recognize excel- 
lence in teaching? 

Are there mechanisms for mentoring new and temporary faculty members 
in the art of teaching and in local customs and expectations? 

Are there mechanisms to assist experienced faculty members who wish to 
improve their teaching? 

If graduate students teach in the program, are they well trained and well 
supervised? 
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■ How does the program evaluate teaching? 

Are teaching dossiers, peer evaluations, student evaluations, and reviews of 
syllabi included in the evaluation process? 

. Are teaching evaluations taken into account in making teaching 
assignments? 

In promotion and tenure decisions? 

Are prcKedures in place to provide counseling tor faculty members whose 
teaching is recognized to be effective tor many students? 

■ \X/hoin the program is cognizant of research in teaching and learning? 

Are these individuals a resource for other faculty members? 
Are there regular occasions (such as faculty meetings, seminars, or brown- 
bag lunches) for discussing teaching strategies and teaching issues? 

■ Do taculty members in the program utilize a variety of teaching techniques? 

Are there opportunities tor collaborative learning, supervised peer teaching, 
and independent study? 

■ How, and how often, do students receive teedback in their courses? 

. What form does the feedback take? 

Are there opportunities tor students to revise and resubmit their work? 



h is. . . useful to think about the major in terms of the appropriately social 
metaphor of "home". . . . [Tjhe maior program provides a "home' for learn- 
ing: a community of peers with whom students can undertake collabora' 
tive inquiries and a faculty charged to care about students* intellectual and 
personal explorations as well as their maturation. 



Advising must be more than monitoring students to ensure that they are making 
satisfactory progress toward a degree, and it must be more than suggesting choices 
among possible options. Quality advising includes discussing goals and expecta* 
tions: of the program, ot the institution, and of the student. It includes discussing 
opportunities in the field and strategies for achieving students' goals both during 
and after their program of study. It includes discussing the relationship among 
courses in the program and between the program and general education. 

High-quality advising is built upon knowing each student's background, be- 
liefs, hopes, and expectations. Ideally, advising is an irreractive process that 
makes a definite contribution to the student's education, ;nforms taculty mem- 
bers about students' concerns, and results in a shared under: tanding of a plan 
or plans of action that will serve well the needs of the student* 



ADVISING 



The Challenge of Connecting Learning, 4 
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■ How are students in the program advised? 

: Does each student meet on a regular basis with a faculty member to discuss 
the student's plans and progress? 

: ■ Do these discussions include attention to questions that transcend the re- 
quirements of the program (such as the rationale for the program, the culture 
of the field, the totality of the undergraduate experience, and student goals 
and expectations)? 

■ Are there also less formal opportunities for faculty/student interaction? 

Are there student clubs, seminars, or interest groups associated with the 
program? 

i Do program faculty members support these activities? 

■ Is advising valued in the program? 

] Are all faculty members expected to participate? 
; ' If advising is the responsibility of only a tew faculty members, are these tew 
provided with sufficient time for regular oneonone interactions with 
students? 

; Is the knowledge that advisers gain about the characteristics of students 
shared with all faculty members in the program? 

; Does it influence the way the program is structured and the way the courses 
are taught? 



INCLUSIVENESS 

What is required of each institution and each field is a strong a//irmation 
of the educational benefits of diversity and a continuing /acwltji dialogue 
about the ways initiator^i [and continuing] experiences in a field can con- 
tribute to. and lay the foundations for, the widest range of students to 
achieve success. 

The Challenge of Connecting Learning, 18 

It no longer is enough to make simple adjustments to accommodate students 
who traditionally have been discouraged from study in a field. Institutions and 
programs must rethink the way they fijnction relative to these students and un- 
dertake comprehensive and sustained change efforts. Plans for change, to be ef- 
fective, must include assessment and accountability measures 
■ >X^at are the program*s goals for enhancing diversity? 

; What do institutional data indicate about the entry and graduation rates ot 

groups of students underrepresented in the field? 
What are the plans to correct any disparities in recruitment and retention of 

different groups? 

o .12 
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Are the current accountability measures successful in ensuring progress in 
attaining goals? 

Who is assigned responsibility tor monitoring and reporting on progress? 
a Does the program assess which courses critically influence students' decisions 
on whether or not to major in the program? 

Do these courses enroll a reasonable percentage of students from groups 
underrepresented in the field? 

Do these students stay in the major at the same rate as other students? 

If these students leave the program in large numbers, why do they leave? 

■ What efforts have been made to identify and remove barriers— such as a climate 
that communicates to underrepresented groups that their participation is not 
welcome, that their success in the field is unlikely, or that the field does not value 
their perspectives and experiences-that may impede entry or success in the pro- 
gram by specific groups of students? 

Have issues of diverse goals, expectations, learning styles, and experiences 
been considered in planning and implementing courses in the program? 

Does the program sponsor workshops for faculty members on climate is- 
sues, including topics such as the dynamics of difference in the classroom, 
stereotyping and bias, or teaching and learning styles? 

■ Is the faculty of the program diverse, representing appropriately the availability 
of faculty members from underrepresented groups? 

If not, has the program evaluated and addressed inequities in recruiting, hir- 
ing, promotion, tenure, and salary of faculty members? 
Has the program evaluatal policies and procedures for disparate impact? 

■ If an imbalance persists after extensive efforts have been made to recruit and 
retain faculty members from underrepresented groups, does the program none- 
theless demonstrate to students diversity among professionals in the field, both 
in perspectives on critical issues in the field and in sex, race, and ethnicity? 

■ Have faculty members in the program explored the potential benefit of alliances 
with units on campus that serve diverse groups of students, such as American eth- 
nic studies, women's studies, minority student affairs, or the women's center? 

Does the program sponsor events that highlight the contributions to the 
field of people of diverse cultures and characteristics? 

Does the program sponsor clubs; student focus groups; and advising, men- 
toring, and mediating services for diverse groups of students? 

■ Where appropriate, does the program assess its curriculum for the inclusion of 
relevant new scholarship about wi^men and minorities? 

Do faculty members participate in seminars, workshops, or professional 
meetings to develop expertise in this material? 
Have efforts to incorporate new scholarship in their courses been successful? 

^'3 
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INSTITUTIONAL SUPPORT 

Collegia! leadership. deserves and requires full institutional support 
from presidents and academic administrators. What are needed in the 
long run are institutional environments that build a sense of common en- 
terprise and institutional priorities that recognize the integral connections 
between nark in the major and overarching goals for liberal learning. 

The Challenge OF Connecting Learning, 21 

Stront; programs are supported by chairs, deans, provosts, and presidents who 
value the teaching activities ot their faculty members. Administrative support 
tor teaching includes more than access to copy machines, well-equipped class- 
rooms, and laboratory and library facilities. It includes support for course and 
curriculum development, faculty development, course and teacher evaluation, 
assessment, and travel to meetings and conferences that tbcus on curriculum or 
teaching. Ic includes promotion and tenure policies that recognize the impor- 
tance of excellent teaching, advising, research in teaching and learning, and cre- 
ative activities that enhance quality teaching. In concert with program reviews, 
administrators at all levels should examine critically their own contributions to 
creating an institutional environment that is supportive of good teaching. 

■ What is the track record of the program and the institution in encouraging, re- 
warding, and promoting excellent teaching? 

What mechanisms are utilized for these purposes? 

Do faculty members describe the atmosphere as one that values teaching 
and teaching-related activities? 

Is there evidence that teaching excellence is an important factor in promt> 
tion and tenure decisions? 

■ How do the institution and the program orient new faculty members? 

Do the orientations include attention to teaching? 

■ Are faculty development activities available for faculty members at all levels? 

Are ongoing workshops or seminars on teaching provided for teaching 
assistants? 

■ What curriculum development activities have been undertaken recently in the 
program? 

Were they adequately supported? 

Have they resulted in documentable improvements in the program? 

■ Are funds available to support assessment and evaluation activities? 

Are these funds well spent? 

■ Is there evidence that research in teaching and learning, course development, 
advising, and other teaching-related activities are valued in the program and in 
the institution? 

:ii 
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OUTCOMES ASSESSMENT 

Corporate attention to the ways in which a major works as a total pro- 
gram, rather than as a set of discrete (and often disparate) courses, can es- 
tablish a series of points at which faculty members both assess and support 
students* continuing progress in the field. 

The Challenge OF Connecting Learning, 18-19 

Every academic program is designed to achieve certain fjoals. Ultimately, the 
quality of the program must be judged on how well it achieves those goals. Fac- 
ulty members' perceptions and anecdotal evidence are not sufficient; evidence on 
the quality of the teaching process is not sufficient. What is necessary is evidence 
that the intended outcomes of the program are realized. The key questions are: 
What are the goals, or intended outcomes, of the program? To what extent are 
these being achieved? 

Strong programs have designed assessment processes to answer these ques* 
tions. These processes judge the outcomes of the program as a whole rather than 
the outcomes of a single course or the performance of an individual instructor or 
the achievements of particular students. Assessment is intended to document 
the successes of the program; flag areas in which goals are not being met satisfac- 
torily; and, ultimately, lead to improvements in the program. Ideally, many of 
the assessment processes also will have direct educational value for students. 
II What are the intended educational outcomes of the program? 

; What processes are in place for measuring the achievements of these 
outcomes? 

j Do the processes provide several kinds of information about student learn- 
ing and achievement? 

Do these processes reflect faculty discussion and decisions about the kinds 
of evidence appropriate to their program's goals, strengths, and emphases? 

■ Are the assessment procedures adopted by the program linked to program 
goals and curricular priorities? 

Do faculty members periodically discuss the results of assessment in relation 
to program goals? 

Do they use assessment evidence in making judgments about curriculum de- 
velopment and revisions? 

■ Does assessment provide opponunities for students to reflect on their progress 
to the program? To integrate different parts of their learning? 

; Do students who take part in assessment activities receive feedback on their 
performance? 

, Is there a culture that invites students to take assessment seriously as a mile- 
stone in their learning and intellecuial development? 
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■ If assessment examinations and assignments are locally developed, are faculty 
members given release time or other compensation to design them? 

; Who is involved in making judgments about the outcomes of assessment? 
; ; Who uses the results? 

; Do faculiy members in the program confer with peers in comparable pro- 
grams in reviewing the outcomes of assessment activities? 

Are there opportunities for students to discuss assessment outcomes in rela- 
tion to their experience in the program? 

■ Tc^ what exten'' are intended outcomes achieved? 

To the extent that the intellectual outcomes are not achieved, what changes 
are being made either in the goals of the program or in the program itself? 

CONCLUSION 

This handbook suggests a general framework through which to explore the effec- 
tiveness of educational programs in the major. Those consulting it will want to 
adapt its specific questions to the particular concerns of their institutions and 
programs. 

As they do this adaptation, and as they implement program review focused 
on educational quality, they should keep at the fore the goal of providing the best 
possible educational experience for the students served. Connecting the pro- 
gram review-as well as the program— to the needs, abilities, aspirations, experi- 
ences, and prior knowledge of students— as well as to the goals and expectations 
of the feculty-will provide the best opponunity tor a program review to lead to 
improved programs. This must be the ultimate goal. 

As faculty members and academic administrators work together to connect 
issues addressed here to their own programs and students, they shape the colle- 
gial dialogue fundamental to the effectiveness of any educational program. 
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and Learnmg, No. 47. San Francisco: Jossey-Bass, 1991. 
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Johnson Foundation. Seven Principles for Good Practice in Undergraduate 
Education. Racine, Wise: The Johnson Foundation, 1989. 
To request a copy of Seven Principles and the accompanying faculty and insti* 
tutional inventories, write to: The Jot ;on Foundation, P.O. Box 547, 
Racine, Wise. 53401-0547. 
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Resources for program review and assessment 

The readings listed below provide helpful perspectives on the historical and in- 
stitutional contexts of the practices of program review and assessment and ofter 
useful suggestions tor different approaches to exploring students' learning and 
evaluating program quality. 

The literature on program review, however, often retlects the major's "ethos 
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report on the major. The Challenge of Connecting Learning. We invite those con- 
cerned with program review to bear in mind the argument ot this handbook: 
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Jossey-Bass, 1991. 
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Education." Revieu'o/ Research in Education 17. Washington: American 
Educational Research AsstKiation, 1991. 

Forrest, Aubrey. Time Will Tell: PortfolioAssisted Assessment for General 

Education, Washington: American Association for Higher Education, 1990. 
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and porttolio assessment. 

Program review in the disciplines 

The guidelines tor educational quality review in this handbook are designed to 
supplement traditional approaches to program review and to give central atten- 
tion to the quality of student learning. For specific fields, this handbook may be 
used to complement disciplinary guidelines. 

Review guidelines for specific fields are available from the following learned 
societies: 

American Association ot Physics Teachers 

5112 Berwyn Road 

College Park, MD 207404100 

American Chemical Society' 
Committee on Professional Training 
1155 16th Street, N.W. 
Washington, D.C. 20036 

American Institute of Biological Sciences 
73011th Street, N.W. 
Washington, D.C. 20001-4584 

American Political Science Association 
1 527 New Hampshire Avenue, N.W. 
Washington, D.C. 20036 
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American Sociological Association 
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1722 N Street, N.W. 
Washington, D.C. 20036 

Association of Departments ot English 

10 Astor Place 

New York NY 10003 




. ' -^^t TTie Association of American Colleges is the only ihsritiitions^ membership higher 
' -^ucation association whose primary mission is imp^nmng lindergradiia^ liberal ed' 
,V uc„don. AAC*s goals-promoting tffelong humane and liberal learning, strengthening 
; ^ iiudtutions of higher education as settings; for liberal learning, ahd^ending the ben- 
efits of liberal leamiag to all-are carried out through research, pr^ectSt publications, 
;-."^^^^::v-1uidw^shop8. ^ 
, . ; ; AAC's programs reflect its commiim^ 

; '^i^i^ ifcral leamtng,'strengd\ening general and specUklfw) ciii » |cu^, im^oinng inching and 
' ' >^^' learning, incrwsing opportunities forequity and achieymcm^^^ insti- 
; f tutional and academic leadership. Founded in . 191 5, AAC'cpm|2^ more than 640 



